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SPECIFICATION 
IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, 
PROGRAM THEREOF, AND RECORDING MEDIUM 



5 TECHNICAL FIELD 

The present invention relates to an image processing apparatus that 
generates one still image from multiple images and a corresponding image 
processing method, as well as a computer program and a recording medium that 
actualize the image processing method. 

10 

BACKGROUND ART 

Image processing apparatuses and image processing methods have been 
proposed to combine multiple images and generate one high-resolution image. 
One known technique selects one scene from a moving image taken with a digital 

15 video camera and generates a higher-resolution still image (having a higher pixel 
density) than an original frame image of the selected scene. This prior art 
technique selects one frame image among (n+l) consecutive frame images as a 
reference image, calculates motion vectors of the other n frame images (subject 
images) relative to the selected reference image, and combines the (n+l) frame 

20 images based on the calculated motion vectors to generate one still image (see, for 
example, Japanese Patent Laying-Open Gazette No. 2000-244851). This image 
processing technique enhances the picture quality and increases the sharpness of 
the composite image, compared with the simple conversion of the resolution of 
one frame image. 

25 

DISCLOSURE OF THE INVENTION 

The prior art image processing technique may, however, not always 
succeed in generating a high-resolution still image, while consuming a relatively 
long time for the image composition. For example, when one subject frame 
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image has blur, the time-consuming image composition often gives only a still 
image of a relatively low resolution. 

The object of the invention is thus to eliminate the drawbacks of the prior 
art technique and to ensure efficient image processing to generate one 
5 high-resolution image from multiple images. 

In order to attain at least part of the above and the other related objects, 
the present invention is directed to an image processing apparatus that generates 
one still image having a high pixel density from multiple images. The image 
processing apparatus includes 1 an image extraction module that extracts the 

10 multiple images used for generation of the one still image! a deviation 
computation module that computes a degree of deviation between each 
combination of the extracted multiple images; an image selection module that 
selects at least two images among the extracted multiple images, based on the 
computed degrees of deviation; and an image composition module that combines 

15 the at least two selected images to generate the one still image. 

There is an image processing method corresponding to the image 
processing apparatus. The image processing method generates one still image 
having a high pixel density from multiple images. The image processing method 
extracts the multiple images used for generation of the one still image, computes 

20 a degree of deviation between each combination of the extracted multiple images, 
selects at least two images among the extracted multiple images based on the 
computed degrees of deviation, and combines the at least two selected images to 
generate the one still image. 

The image processing apparatus and the corresponding image processing 

25 method (hereafter collectively referred to as the image processing) computes the 
degree of deviation between each combination of the extracted multiple images 
and selects at least two images based on the computed degrees of deviation, prior 
to the actual image composition. The at least two images used for the image 
composition are selected according to the computed degrees of deviation. This 



2 



arrangement of the invention ensures efficient processing for image composition. 

In one preferable embodiment of the invention, the image processing 
specifies a reference image as a base of composition of the one still image and 
extracts the multiple images in a correlated order with the specified reference 
5 image. 

The image processing of this embodiment extracts the multiple images, 
which are used to generate the one still image having the high pixel density, in 
the corrected order with the specified reference image. The multiple images 
used for the image composition are automatically extracted by specification of one 

10 reference image. This arrangement does not require specification of all the 
multiple images used for the image composition and simplifies the user's 
operation for image composition. 

In the image processing of the invention, the multiple images may be 
consecutively arranged in time series, and the correlated order may be a time 

15 series order from the specified reference image. The multiple images may be 
extracted in various orders, for example, in an alternate order before and after 
the specified reference image or in an alternate order in time series from the 
specified reference image. Extraction of the images consecutively arranged in 
time series from the specified reference image desirably simplifies the image 

20 extraction process. 

The image processing of the invention may display the number of images 
used for the image composition, prior to generation of the one still image. This 
arrangement enables the user to readily recognize the number of images actually 
used for the image composition. One possible modification may display the 

25 number of images excluded from the objects of image composition, in addition to 
or in place of the number of images used for the image composition. 

The image processing of the invention may give an alarm when the 
number of the at least two selected images does not reach a preset minimal 
number, or may select either execution or non-execution of the image composition 
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when the number of the at least two selected images does not reach a preset 
minimal number. 

The first arrangement informs the user of the possibility that the 
generated one still image has insufficient sharpness since the number of images 
5 used for the image composition does not reach the preset minimal number. The 
second arrangement enables the user to select execution or non-execution of the 
image composition when the number of images used for the image composition 
does not reach the preset minimal number. 

The image processing of the invention may discontinue the image 
10 composition when the number of the at least two selected images does not reach a 
preset minimal number. 

Exclusion of the image having the computed degree of deviation out of a 
preset threshold range may decrease the number of images used for image 
composition to be less than the arbitrarily set minimal number, for example, two 
15 images or three images. The image processing of this arrangement discontinues 
the image composition in such cases. When generation of the still image having 
the high pixel density is not expected, the time-consuming image composition 
may thus be discontinued at an early stage. For example, setting a certain value 
to the minimal number enables immediate discontinuation of the image 
20 composition when even one of the computed degrees of deviation is out of the 
preset threshold range. 

In another preferable embodiment of the invention, the image processing 
excludes any image having the computed degree of deviation out of a preset 
threshold range from the extracted multiple images, and selects images other 
25 than the image excluded by said exclusion module as the at least two images. 
Comparison between the computed degree of deviation and the preset threshold 
range enables the user to readily identify the image unsuitable for the image 
composition. 

The computed degree of deviation may be at least either of a translational 
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deviation between two images in a translational direction and a rotational 
deviation between the two images in a rotational direction. Any image having at 
least either of the translational deviation and the rotational deviation out of the 
preset threshold range may be excluded from the objects of image composition. 
5 In the image processing of the invention, the preset threshold range may 

be set as specific values or may be expressed by a rate to a total number of pixels 
constituting the resulting still image. For example, the preset threshold range 
may be several to ten % of the number of vertical pixels and of the number of 
horizontal pixels. The preset threshold range may be fixed values. For 

10 example, the preset threshold range of the translational deviation may be ±16 
pixels and the preset threshold range of the rotational deviation may be ±1°. 
Any image having at least either of the translational deviation out of ±16 pixels 
and the rotational deviation out of ±1° is excluded from the objects of image 
composition or causes discontinuation of the image composition. When the 

15 translational deviation between two consecutive images in time series is out of 
this threshold range, it is highly probable that one of the consecutive images has 
blur. Based on the result of the comparison between the translational deviation 
and the preset threshold range, any image expected to make substantially no 
contribution to the result of image composition is thus excluded from the objects 

20 of image composition or causes discontinuation of image composition. 

In one application of the image processing of the invention, the multiple 
images may be multiple frame images included in a moving image. The 
technique of the invention is especially effective to combine multiple 
low-resolution frame images and generate one high-resolution still image. 

25 In another application of the image processing of the invention, the 

multiple images may be multiple still images having information of an exposure 
time, which varies according to lightness of a photographic subject at a shooting 
time. The threshold range may be set for each still image, based on the varying 
exposure time. 
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This application sets the threshold range of the degree of deviation for 
each still image, based on the information of the exposure time of the still image 
at the shooting time. The concrete procedure calculates an allowable degree of 
deviation within the exposure time of one still image from the rate of the 
5 exposure time to the shooting time interval between two subject images, and sets 
the threshold range corresponding to the calculated allowable degree of deviation. 
In this application, the threshold range is not uniformly fixed but is adequately 
varied corresponding to each still image. 

The technique of the invention may also be actualized by a computer 
10 program having any of the above arrangements and a recording medium in which 
such a computer program is recorded. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 schematically illustrates the configuration of an image processing 
15 system in a first embodiment of the invention; 

Fig. 2 is a flowchart showing an image processing routine executed in the 
first embodiment; 

Fig. 3 shows positional deviation between two images; 

Fig. 4 shows a process of computing translational deviations by the 
20 gradient method; 

Fig. 5 schematically shows a rotational deviation of a pixel; 
Fig. 6 shows a range contributing the increased sharpness of a resulting 
composite image relative to the computed degree of deviation; 

Fig. 7 is a flowchart showing an image processing routine executed in a 
25 second embodiment of the invention; 

Fig. 8 shows a process of computing a degree of deviation in the second 
embodiment; 

Fig. 9 is a flowchart showing an image processing routine executed in a 
third embodiment of the invention; and 
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Fig. 10 shows the exposure time relative to the shooting period of still 
images taken in a continuous shooting mode. 

BEST MODES OF CARRYING OUT THE INVENTION 

5 Some modes of carrying out the invention are described below in the 

following sequence as preferred embodiments* 

A. First Embodiment 

Al. Configuration of Image Processing Apparatus 
A2. Image Processing 
10 A3. Computation of Deviation 

A4. Setting Thresholds 

B. Second Embodiment 

Bl. Image Processing 

C. Third Embodiment 

15 CI. Configuration of Image Processing Apparatus 

C2. Image Processing 
C3. Setting Thresholds 

A. First Embodiment 

20 Al. Configuration of Image Processing Apparatus 

Fig. 1 schematically illustrates the configuration of an image processing 
system 100 in a first embodiment of the invention. As illustrated, the image 
processing system 100 includes an image database 20 that supplies image data of 
moving images and still images, a personal computer 30 that functions as an 

25 image processing apparatus and performs series of image processing to process 
multiple images supplied from the image database 20, a user interface 40 that 
enables a user to give instruction for performing the series of image processing, 
and a color printer 50 that outputs a resulting processed image. 

The image database 20 includes imaging devices and image storage 



7 



devices that are capable of supplying image data to the personal computer 30. 
In the system of this embodiment, the image database 20 includes a digital video 
camera 21, a digital still camera 22, a DVD 23, and a hard disk 24. The image 
database 20 of the first embodiment stores moving image data taken with the 
digital video camera 21. In the description below, images processed by the image 
processing of this embodiment are referred to as frame images. The frame 
images represent multiple consecutive images sequentially arranged in time 
series and are part of a moving image. 

The personal computer 30 includes a CPU 31 that executes the series of 
image processing, a ROM 32, a RAM 33, a hard disk 34 that stores an image 
processing software program installed therein, and an I/F circuit 35 that 
communicates with external devices, that is, the image database 20, the user 
interface 40, and the color printer 50. The image processing software program 
installed in the hard disk 34 is executed to combine multiple input frame images 
and generate one high-resolution still image. The personal computer 30 with 
the installed image processing software program comprehensively works as the 
image processing apparatus and has the functions of the f image extraction 
module', the 'deviation computation module', the 'exclusion module', and the 
'image composition module 1 of the invention. The flow of image processing will 
be described later in detail. 

The user interface 40 includes a keyboard 41 and a mouse 42 that are 
manipulated by the user to perform the series of image processing, and a display 
43 that displays multiple frame images before the image processing and a 
resulting still image as a result of image composition. 

A2. Image Processing 

Fig. 2 is a flowchart showing an image processing routine executed in the 
first embodiment to combine multiple image data and generate one still image. 
In the image processing system 100 having the hardware configuration discussed 
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above, the personal computer 30 starts the installed image processing software 
program, in response to the user's operation of the keyboard 41. 

On a start of image processing, the personal computer 30 inputs moving 
image data as a set of frame image data from the image database 20 and plays 
5 back the input moving image data on the display 43. The user activates the 
suspend playback function to freeze the reproduced moving image at the user's 
desired scene for output of a still image. The personal computer 30 specifies the 
frozen scene (frame image) in response to the user's selection (step S200). 

The personal computer 30 extracts the specified frame image and 

10 multiple consecutive frame images in time series after the specified frame image 
to be used for the subsequent image processing (step S210). The procedure of 
this embodiment extracts four consecutive frame images in time series from the 
user's operation timing for specifying the desired frame image. Among the four 
frame images, the user's specified frame image (that is, the first frame image in 

15 time series) is referred to as reference frame image Fl, and the other three frame 
images are referred to as subject frame images F2 to F4. The user may 
arbitrarily set the number of frame images to be extracted. 

The personal computer 30 then computes the degree of deviation between 
each pair of the extracted four frame images (step S220). The degree of 

20 deviation computed at step S220 is a 'positional deviation' between two images as 
shown in Fig. 3. The positional deviation is defined by three factors, 
translational deviations 'u' and V in translational directions and a rotational 
deviation '5' in a rotational direction. The personal computer 30 computes the 
degree of deviation (u,v,5) between every combination of the reference frame 

25 image Fl and the three subject frame images F2 to F4. The method of 
computing the degree of deviation will be described later in detail. 

The personal computer 30 determines whether the computed degree of 
deviation between every combination of the four adjoining frame images in time 
series is within a predetermined range (defined by preset thresholds). For 
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example, when the computed degree of deviation between the subject frame 
images F2 and F3 is out of the preset thresholds, an intentional quick pan of the 
digital video camera 21 is assumed between the subject frame images F2 and F3. 
It is thus highly probable that the subject frame image F3 has blur due to the 
quick camera pan. The personal computer 30 then excludes the subject frame 
image F3 having the high possibility of blur from objects of image composition 
(step S230). 

The procedure of the embodiment sets a lower limit of the translational 
deviations V and V and compares the translational deviations 'u' and V between 
every combination of the extracted four frame images with the lower limit. 
When both the degree of deviation between the adjoining subject frame images F2 
and F3 and the degree of deviation between the adjoining subject frame images 
F3 and F4 are within the predetermined range, the subject frame images F2 and 
F4 may be substantially identical with each other. In such cases, these subject 
frame images F2 and F4 are not required simultaneously for image composition. 
Namely when two frame images have no significant deviation and are assumed to 
be identical, only one of the two frame images is sufficient for the image 
composition. The procedure of this embodiment sets 0.1 pixel to the lower limit 
of the translational deviations 'u ? and V between two frame images and excludes 
one of the two frame images from the objects of image composition when the 
computed translational deviations 'u' and V are not greater than 0.1 pixel. 

The personal computer 30 uses the non-excluded remaining subject frame 
images and the reference frame image Fl to implement image composition and 
generate a composite image (step S240). The concrete procedure of image 
composition lays a corrected subject frame image (for example, the subject frame 
image F2) after correction of the positional deviation over the reference frame 
image Fl and determines tone values of the respective pixels included in a 
resulting composite image from the tone values of the respective pixels in the 
reference frame image Fl and the subject frame image F2. The tone values of 
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the respective pixels in the resulting composite image are determined by the 
known bilinear method, although another known method, such as the nearest 
neighbor method or the bicubic method may be adopted instead of the bilinear 
method. The personal computer 30 sequentially executes the image composition 
by the bilinear method to complete one composite still image. 

The exclusion of the subject frame images based on the results of 
comparison with the preset thresholds may decrease the total number of the 
subject frame images used for the image composition. Even in such cases, the 
procedure of this embodiment performs the image composition with the 
remaining subject frame images and the reference frame image. The personal 
computer 30 displays the number of frame images eventually used for the image 
position on the display 43. The number of frame images used for the image 
composition may be displayed prior to execution of the image composition. One 
modified procedure may further extract a subsequent consecutive frame images 
(a fifth frame image) in time series from the image database 20 as an additional 
subject frame image and repeat the processing of and after step S220. This 
modified procedure keeps the fixed number of the subject frame images used for 
the image composition. Another possible modification may extract a previous 
frame image before the reference frame image Fl in time series as an additional 
subject frame image. 

The personal computer 30 displays the composite still image thus 
obtained in a selected position on the display 43 and concludes the image 
processing routine. The composite still image may be output to the color printer 
20 or the hard disk 34 in response to the user's request. 

The series of image processing excludes subject frame images making 
substantially no contribution to resolution enhancement and increased sharpness 
of a resulting composite image, for example, a subject frame image having a high 
possibility of blur and a subject frame image substantially identical with the 
reference frame image Fl, from the objects of image composition, prior to 
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execution of the image composition. This arrangement ensures efficient image 
processing. 

A3. Computation of Deviation 
5 As shown in Fig. 3, the positional deviation between the reference frame 

image Fl and the subject frame image F2 is expressed by the three parameters 
(u,v,5). The reference frame image Fl is expressed in an orthogonal coordinate 
system (xl,yl), where the center of the image is on the origin and xl and yl 
respectively denote a lateral axis and a vertical axis. The subject frame image 

10 F2 is expressed in an orthogonal coordinate system (x2,y2), where the center of 
the image is on the origin and x2 and y2 respectively denote a lateral axis and a 
vertical axis. In the illustrated example of Fig. 3, the subject frame image F2 
has a translational deviation V in the lateral direction and a translational 
deviation V in the vertical direction relative to the reference frame image Fl and 

15 a rotational deviation '5' in the rotational direction about the center of the subject 
frame image F2. 

The procedure of this embodiment adopts the gradient method, which 
uses luminance values of the respective pixels between each frame images and 
estimates a pixel position in a smaller unit than one pixel. Fig. 4 shows a 

20 process of computing translational deviations by the gradient method. Figs. 4(a) 
and 4(c) show luminance values of pixels on the respective images, and Figs. 4(b) 
and 4(d) show the principle of the gradient method. A certain pixel on the 
reference frame image Fl has coordinates (xli,yli) and a luminance value 
Bl(xli,yli). A selected pixel having coordinates (x2i,y2i) on the subject frame 

25 image F2 is located between the coordinates (xli,yli) and coordinates 
(xli+l,yli+l) on the reference frame image Fl. The coordinates of the selected 
pixel on the reference frame image Fl are expressed as (xli+Ax, yll+Ay). 

As shown in Figs. 4(b) and 4(d), when the pixel having the coordinates 
(x2i,y2i) on the subject frame image F2 is assumed to be located at coordinates 
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(xli+Ax,yli) on the reference frame image Fl and Px is defined as Equation (l), 
Equation (2) given below is obtained* 



Px = Bl(xli+l,yli) - Bl(xli,yli) 
PxAx = B2(x2i,y2i) - Bl(xli,yli) 



(1) 



(2) 



Equation (2) is written as Equation (3), where Bl(xli,yli) and B2(x2i,y2i) are 
simply expressed as Bl and B2- 



Determination of Ax satisfying Equation (3) gives a translational deviation of the 
subject frame image F2 in the x-axis direction. The concrete procedure 
computes Ax of the respective pixels in the subject frame image F2 and averages 
the computed values Ax as the translational deviation of the subject frame image 
F2 in the x-axis direction. 

Similarly when the pixel having the coordinates (x2i,y2i) on the subject 
frame image F2 is assumed to be located at coordinates (xli,yli+Ay) on the 
reference frame image Fl and Px is defined as Equation (4), Equation (5) given 
below is obtained* 



Equation (5) is written as Equation (6), where Bl(xli,yli) and B2(x2i,y2i) are 
simply expressed as Bl and B2* 



{Px'Ax-(B2-Bl)} 2 = 0 



(3) 



Py = Bl(xli,yli+l) - Bl(xli,yli) 
Py-Ay = B2(x2i,y2i) - Bl(xli,yli) 



(4) 



(5) 



{Py-Ay -(B2-B1)}2 = 0 



(6) 
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Determination of Ay satisfying Equation (6) gives a translational deviation of the 
subject frame image F2 in the y-axis direction. The concrete procedure 
computes Ay of the respective pixels in the subject frame image F2 and averages 
the computed values Ay as the translational deviation of the subject frame image 
F2 in the y-axis direction. 

Equation (3) regards only the x-axis direction and Equation (6) regards 
only the y-axis direction. Expansion of these equations to take into account both 
the x-axis direction and the y-axis direction gives" 



The values of Ax and Ay that minimize Equation (7) are determined by the least 
squares method as the translational deviations *u' and V. 

The above calculation is on the assumption that there is only a 
translational deviation between two frame images. The following description 
regards computation of the degree of deviation by additionally taking into 
account the rotational deviation 5. Fig. 5 schematically shows a rotational 
deviation of a pixel. A distance V of coordinates (xl,yl) from the origin O on the 
reference frame image Fl and a rotational angle '9' of the pixel from the xl axis 
are defined as* 



In the illustrated example of Fig. 5, for the simplicity of explanation, the frame 
image has its center on the origin O and rotates about the origin O. 

The rotation about the origin O shifts the coordinates (xl,yl) on the 
reference frame image to (x2,y2), which are identical with the coordinates (x2,y2) 
on the subject frame image. A displacement Ax in the x*axis direction and a 



S 2 = Z{Px Ax + Py Ay - (B2-Bl)}2 



(7) 



r = ( X 2 + y2)i/2 
0 = tanKx/y) 



(8) 



(9) 



14 



displacement Ay in the y-axis direction by this rotation are defined by- 

Ax = x2 - xl « -r-5sin5 = —5-yl (10) 
Ay = y2 - yl « r-5-cos5 = 5 x1 (ll) 

5 

On the assumption that the rotational deviation '5' is a minimal amount, the 
approximate expressions cos8 » 1 and sin6 » 5 are applied. 

By taking into account the rotational deviation '5' in addition to the 
translational deviations V and V, the displacements Ax and Ay in Equation (7) 
10 are expressed as- 

Ax = u -5-yl (12) 
Ay = v + 5x1 (13) 

15 Substitution of the displacements into Equation (7) gives a general equation of 

S 2 = E{Px-(u - Sy) + Py.(v + 5-x) - (B2-B1)} 2 (14) 

Determination of the deviations ? u f , V, and 5 that minimize S 2 of Equation 
20 (14) by the least squares method accurately computes the degree of deviation of 
less than 1 pixel between two frame images. The embodiment adopts the 
gradient method for computation of the degree of deviation. Another technique, 
for example, the block matching method, the conjugate gradient method, and 
their combination, may be applied to compute the degree of deviation. 

25 

A4. Setting Thresholds 

The range (thresholds) contributing the resolution enhancement and the 
increased sharpness of a resulting composite image is set with regard to the 
degree of deviation (u,v,8) computed as discussed above. Fig. 6 shows a range 

15 



contributing the increased sharpness of a resulting composite image relative to 
the computed degree of deviation. The abscissa and the ordinate respectively 
show the translational deviations 'u f and V and the rotational deviation '5', and 
areas (a), (b), and (c) are set to specify the range contributing the resolution 
enhancement and the increased sharpness. The area (a) represents a range of 
positional deviation between adjoining frame images caused by general hand 
movement or camera pan. The area (b) represents a range of positional 
deviation between adjoining frame images caused by a quick camera can or an 
intentional camera rotation. The area (c) represents a possible range of the 
translational deviations and the rotational deviation. 

In this embodiment, the area (a) is set to satisfy conditions of —16 pixels < 
translational deviations 'u\V < 16 pixels and —1° < rotational deviation '6' < 1°. 
When the computed degree of deviation between a subject frame image and the 
reference frame image is included in the area (a), the subject frame image is 
identified to contribute the increased sharpness of the resulting image. Namely 
±16 pixels and ±1° are set as thresholds contributing the increased sharpness of 
the resulting image. The thresholds are fixed values in this embodiment, but 
may be varying values according to the conditions of a resulting composite image 
and the conditions of image composition. For example, the thresholds may be 
set as ratios (for example, several % or ten and several %) to the number of the 
horizontal pixels and to the number of the vertical pixels included in a resulting 
composite image. In another example, the thresholds may be set according to 
the optical magnification of an original moving image as the extraction source of 
frame images. The low optical magnification generally has a low potential for 
blur. The thresholds may be updated as a result of learning. The thresholds 
may be varied gradually, based on the user's judgment of composite images. The 
thresholds are raised upon the user's approval of the sharpness in a composite 
image, while being lowered upon the user's disapproval of the sharpness. 

In a moving image taken with a conventional digital video camera, the 
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positional deviation between adjoining frame images due to hand movement or 
camera pan typically includes the translational deviations of several pixels and 
the rotational deviation of not greater than 1°. Selection of frame images having 
the computed degrees of deviation within the area (a) effectively excludes subject 
frame images having significant blur, for example, due to a quick camera pan 
from the objects of image composition. Such selection also excludes subject 
frame images across scene transitions from the objects of image composition. 

The rotational deviation '8' of the frame image included in the area (a) is 
within ±1°. This range satisfies the approximate expressions of cosS * 1 and sin5 
« 5 introduced in the process of computing the degree of deviation. Namely this 
range ensures the sufficient computation accuracy of the degree of deviation. 
Selection of subject frame images included in the area (a) as the objects of image 
composition desirably excludes subject frame images that lower the computation 
accuracy of the degree of deviation. 

B. Second Embodiment 
Bl. Image Processing 

Fig. 7 is a flowchart showing an image processing routine executed in a 
second embodiment of the invention to combine multiple image data and generate 
one still image. The primary difference from the image processing of the first 
embodiment shown in the flowchart of Fig. 2 is a decision step for withdrawal of 
image composition. The identical processing steps with those of the first 
embodiment are not described in detail. The image processing system of the 
second embodiment has the hardware components that are identical with those of 
the first embodiment and are expressed by the same numerals and symbols and 
are thus not specifically described here. 

When the image processing routine of this embodiment shown in Fig. 7 is 
triggered by the user's operation of the keyboard 41, like the first embodiment, 
the personal computer 30 first inputs moving image data from the image 
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database 20, specifies a frozen frame image as a reference frame image Fl in 
response to the user's selection (step S400), and extracts three consecutive frame 
images in time series after the specified reference frame image Fl as subject 
frame images F2 to F4 (step S410). 

The personal computer 30 then computes the degree of deviation (u,v,5) 
between each pair of the extracted four frame images (step S420). The deviation 
computation process adopts the method described in the first embodiment to 
compute the degree of deviation (u,v,8) between every combination of the 
reference frame image Fl and the three subject frame images F2 to F4. In the 
description below, the computation of the degree of deviation between the 
reference frame image Fl and the subject frame image F2, the computation 
between the subject frame image F2 and the subject frame image F3, and the 
computation between the subject frame image F3 and the subject frame image F4 
are respectively referred to as process Sl"2, process S2-3, and process S3-4 as 
shown in Fig. 8. 

The personal computer 30 subsequently determines whether the 
computed degree of deviation (u,v,8) between one pair of adjoining frame images 
is within the preset thresholds described in the first embodiment (step S430). 
The concrete procedure first compares the computed degree of deviation (u,v,5) by 
the process Sl-2 with the preset thresholds at step S430. When the computed 
degree of deviation (u,v,5) is out of the preset thresholds at step S430, the 
personal computer 30 displays an alarm message telling that image composition 
may result in an unsharp composite image, on the display 43 (step S460) and 
opens a selection window on the display 43 to enable the user to select 
continuation or discontinuation of image composition (step S465). In response to 
the user's selection of discontinuation of image composition at step S465, the 
personal computer 30 discontinues the image composition (step S470) and 
concludes the image processing routine of Fig. 7. 

When the computed degree of deviation (u,v,5) is within the preset 
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thresholds at step S430 or when the computed degree of deviation (u,v,5) is out of 
the preset thresholds at step S430 but the user selects continuation of image 
composition at step S465, the personal computer 30 determines whether the 
comparison between the computed degree of deviation (u,v,5) and the preset 
thresholds has been completed for all pairs of adjoining frame images (step S440). 
When the comparison between the computed degree of deviation (u,v,5) and the 
preset thresholds has not been completed for all the pairs of adjoining frame 
images, the image processing routine goes back to step S430 to determine 
whether the computed degree of deviation between a next pair of adjoining frame 
images is within the preset thresholds. Upon condition that the computed 
degree of deviation by the process Sl-2 is within the preset thresholds, the 
procedure makes a comparison between the computed degree of deviation by the 
process S2-3 and the preset thresholds. Upon condition that the computed 
degree of deviation by the process S2-3 is within the preset thresholds, the 
procedure makes a comparison between the computed degree of deviation by the 
process S3-4 and the preset thresholds. In the course of this repetition, when 
any of the computed degrees of deviation dissatisfies the threshold condition, the 
personal computer 30 displays the alarm message (step S460) and opens the 
selection window to enable the user to select continuation or discontinuation of 
image composition (step S465). 

When the comparison between the computed degree of deviation (u,v,5) 
and the preset thresholds has been completed for all the pairs of adjoining frame 
images at step S440, the personal computer 30 implements the actual image 
composition (step S450). In the same manner as the first embodiment, the 
image processing routine of the second embodiment compares the translational 
deviations 'u f and V between every combination of the extracted frame images 
with the preset lower limit, prior to the actual image composition. The 
comparison excludes the identical or substantially identical subject frame image 
with another frame image from the objects of image composition. In the same 
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manner as the first embodiment, the personal computer 30 displays a composite 
still image on the display 43 and exits from the series of image processing. 

The image processing of the second embodiment displays the alarm 
message and asks the user to select continuation or discontinuation of image 
composition, when the extracted frame images include any frame image making 
substantially no contribution to the resolution enhancement and the increased 
sharpness of a resulting composite image. This arrangement enables the user to 
readily identify the frame images expected to make a contribution to the 
increased sharpness of the composite image, and to select continuation of 
time-consuming image composition only for the frame images expected to make a 
contribution to the increased sharpness. This ensures efficient image processing. 
In the moving image taken with a digital still camera, it is generally improbable 
that a frame image adjoining to a frame image making contribution to the 
increased sharpness of a resulting composite image has an abrupt positional 
deviation. The procedure of the embodiment displays the alarm message when 
the computed degree of deviation between any pair of the adjoining frame images 
dissatisfies the threshold condition. The user can thus select continuation or 
discontinuation of image position at a relatively early stage of image composition. 
This characteristic enables establishment of an efficient image processing 
system. 

The procedure of the second embodiment computes the degrees of 
deviation between all the pairs of adjoining frame images and then sequentially 
compares the computed degrees of deviation with the preset thresholds. One 
modified procedure may compute the degree of deviation between one pair of 
adjoining frame images and compare the computed degree of deviation with the 
preset thresholds. Only upon satisfaction of the threshold condition, the 
modified procedure computes the degree of deviation between a next pair of 
adjoining frame images. The procedure of the second embodiment displays the 
alarm message and discourages the user from continuing image composition, 
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when any of the computed degrees of deviation is out of the preset thresholds. 
One modified procedure may set an allowable limit number of frame images for 
image composition, and may discontinue the image composition when the 
remaining number of frame images does not reach the allowable limit number. 
5 For example, when one frame image among extracted four frame images has the 
computed degree of deviation out of the preset thresholds, the modified procedure 
excludes this frame image and uses the remaining three frame images to 
implement image composition. When two or more frame images have the 
computed degrees of deviation out of the preset thresholds, the modified 
10 procedure discontinues the image composition. 

C. Third Embodiment 

CI. Configuration of Image Processing Apparatus 

The image processing system in a third embodiment of the invention has 

15 the hardware components that are identical with those of the first embodiment 
and are expressed by the same numerals and symbols and are thus not 
specifically described here. The primary difference from the first embodiment is 
image data taken and stored in the image database 20 (that is, the material of 
image composition) as described below. 

20 The image data treated in the third embodiment are multiple still image 

data taken in a continuous shooting mode of a digital still camera. Each still 
image data is obtained as an Exif image file, which includes JPEG image data 
and required pieces of shooting information, such as the date and time of shooting, 
the exposure time, and use or non-use of flash. In the continuous shooting mode 

25 of the digital still camera of this embodiment, the user's long depression of the 
shutter enables still images to be taken at intervals of 1/30 seconds. The image 
database 20 stores at least four Exif files of still images taken at the intervals of 
1/30 seconds. The image data included in each image file is taken with the 
automatically varying exposure time according to the lightness of a photographic 
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subject. This embodiment treats the still image data taken at the intervals of 
1/30 seconds, but the technique of the invention is also applicable to treat still 
image data taken in a standard continuous shooting mode, for example, at a rate 
of 4 or 9 images per second. 

C2. Image Processing 

Fig. 9 is a flowchart showing an image processing routine executed in the 
third embodiment to combine multiple image data and generate one still image. 
In the image processing system of the above configuration, the image processing 
software program installed in the personal computer 30 is executed in response to 
the user's operation of the keyboard 41. 

The personal computer 30 first reads multiple image files of multiple still 
images, which are taken in the continuous shooting mode, from the image 
database 20 and displays the multiple still images on the display 43. The user 
clicks and selects a desired still image among the still images on the display 43 
with the mouse 42. The personal computer 30 specifies the selected still image 
in response to the user's instruction (step S500). 

The personal computer 30 then extracts three consecutive still images in 
time series immediately after the specified still image out of the read still image 
files (step S510). The personal computer 30 subsequently computes the degree 
of deviation between each pair of the four still images (one specified still image 
and three extracted still images) (step S520) in the same manner as the first 
embodiment. For the purpose of distinction, in the description below, the 
specified still image is referred to as reference image and the extracted still 
images are referred to as subject images. 

The personal computer 30 sets thresholds of the degree of deviation to 
identify the suitability of each subject image for image composition (step S530). 
Unlike the procedure of the first embodiment, the procedure of the third 
embodiment does not use the fixed thresholds but sets the thresholds for each 
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subject image, based on the exposure time included in the shooting information of 
each image file. The method of setting the thresholds will be described later. 

The personal computer 30 determines whether the computed degree of 
deviation between every combination of the four still images is within the preset 
thresholds according to the exposure time. The still images continuously shot 
with the digital still camera generally have significantly small rotational 
deviations '8\ The procedure of this embodiment thus compares only the 
computed translational deviations 'u' and V with the preset thresholds. Based 
on the result of the comparison, the personal computer 30 excludes any subject 
image dissatisfying the threshold condition from the objects of image composition 
(step S540). 

The personal computer 30 uses the non-excluded remaining subject 
images and the reference image to implement image composition and generate a 
composite image (step S550) in the same manner as the first embodiment. The 
personal computer 30 shows the resulting composite still image on the display 43 
and exits from the series of image processing. 

C3. Setting Thresholds 

Fig. 10 shows the exposure time relative to the shooting period of still 
images taken in the continuous shooting mode. Each of the still images Fl, F2, 
F3, ... continuously shot in time series at every shooting period Tf has a varying 
exposure time Tp. In this embodiment, the shooting period Tf is a constant 
interval of 1/30 seconds as mentioned previously, and the exposure time Tp varies 
for each still image. 

The procedure computes the degree of deviation (u,v,8) between the two 
consecutive still images Fl and F2 in time series. In the continuous shooting 
mode giving the significantly small rotational deviatioii '5 1 , a moving distance 'mf 
of the subject between the two adjoining still images is expressed as- 
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mf=(u2 + v 2)i/2 



(15) 



where 'mf denotes a moving distance within the shooting period Tf. 

On the assumption that a moving distance 'mp' of the subject in one still 
image is proportional to the moving distance 'mf of the subject between the two 
adjoining still images, the moving distance 'mp' of the subject in one still image is 
expressed as- 



When the moving distance 'mp' of the subject in one still image exceeds 
one pixel, there is a possibility that the still image has blur. When an allowable 
limit of blur is a predetermined value 'mpt' that is not greater than 1 pixel, the 
moving distance 'mf of the subject between the two adjoining still images is 
expressed as- 



A range satisfying this conditional expression is set to the thresholds of the 
degree of deviation (u,v,5) between two still images. 

The conditional expression (17) gives the narrower allowable range of the 
degree of deviation to still images having the longer exposure time Tp, while 
giving the wider allowable range to still images having the shorter exposure time 
Tp. This conditional expression (17) accordingly enables the adequate 
thresholds of the degree of deviation to be set for each of still images having 
different shooting conditions. Even when the shooting period Tf is not 1/30 
seconds but is changed, setting the thresholds satisfying the conditional 
expression (17) enables selection of still images having a high potential for blur. 

The procedure of this embodiment sets the thresholds for each still image 



mp = mf x Tp / Tf 



(16) 



mf = mpt x Tf /Tp 



(17) 
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based on the exposure time Tp and adequately excludes till images having a high 
potential for blur. This arrangement ensures efficient image processing. The 
procedure of the third embodiment excludes inadequate images from the objects 
of image composition like the first embodiment, but may be modified to enable 
the user to select continuation or discontinuation of image composition like the 
second embodiment. 

The first through the third embodiments of the invention treat the 
consecutive images in time series as the material of image composition. The 
material of image composition is, however, not restricted to the consecutive 
images in times series but may be images taken from an identical angle on 
different dates or images taken simultaneously with two digital video cameras or 
two digital still cameras located near to each other. In the last case, even when 
the two cameras have relatively low resolutions, the image composition enables 
output of a high-resolution still image. 

The embodiments and their modifications discussed above are to be 
considered in all aspects as illustrative and not restrictive. There may be many 
other modifications, changes, and alterations without departing from the scope or 
spirit of the main characteristics of the present invention. 

In the systems of the embodiments described above, the personal 
computer is the image processing apparatus of the invention. The technique of 
the invention may also be attained by diversity of other devices, for example, 
printers, digital video cameras, and digital still cameras. The image processing 
apparatus and the corresponding image processing method of the invention may 
be actualized independently on the computer or may be incorporated in the 
diversity of other devices. 
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